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Abstract 

We consider the branching ratio for the process f ^ 07 in the context of the General Two Higgs 
Doublet Model type III. We find that taking into account reasonable values for the parameter tan (3 
is possible to get values of this branching ratio up to orders of magnitude lying in the range of 
sensitivity of near future top quark experiments. For values of B{t 07) ~ 1 — 9 x 10~^, values 
between 8 — 15 for tan /? are allowed. 
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I. INTRODUCTION 



The search for Flavor Changing Neutral (FCN) processes is one of the most interesting 
possibihties to test the Standard Model (SM), with the potential for either discovering or 
putting stringent bounds on new physics. In the SM, there are not FCN processes at the 
tree-level and at one-loop level they are GIM suppressed |l|]. For this reason, the branching 
ratios predicted within the SM for these kind of processes are in general, very small. Special 
interest has been devoted to the so-called rare top quark decays, e.g. t — > c{j,Z,g) and 
t — > c{h^ , H^, A^) [0. The reason for the interest in these decays is again that the observation 
of a single event of this kind in near future colliders would imply evidence of new physics 
which could be related with the Higgs sector of the theory. 

In the framework of the SM the decay t — » bW is by far the dominant one and the rates 
for the rare decays of the top quark into gauge bosons (7, Z, g) plus c-quark are too much 
small. With the current experimental value of the top quark mass, the ratio for t — > 07 is 
of the order of ~ 5 x 10"^^, for the Z° boson ~ 1 x 10"^^ and for the gluon ~ 4 x 10"^^ 
These numbers are so tiny because in the SM the FCN decays of the top quark are present 
at the one loop-level and they are controlled by down-type quarks and are beside strongly 
suppresed by the GIM mechanism 

But some of this undesirable features can change with physics beyond the SM. For in- 
stance, there is not any fundamental reason for having only one Higgs doublet. In fact, 
for two or more scalar doublets, FCN couplings could be generated at tree level unless and 
ad hoc discrete symmetry is imposed . Then it is possible to enhance the branching ratios 
for rare top quark decays in different kinds of models which include new physics. 

Models like Minimal Supersymmetric Standard Model (MSSM) with R-parity, that relax 
the universality of soft breaking terms with a large flavor mixing only between second and 
third family can reach values such as B{t —>■ C7(Z)) ~ 10"*^ and B{t eg) ~ 10~^ [^. On 
the other hand, it is possible to introduce new couplings with R-parity broken and in this 
context, some of these channels could become observable. For example, very enhanced ratios 
are predicted in this framework such as B(t — > 07) ~ 10^^, B(t cZ) ~ 10~^ and B(t — >■ 
eg) ~ 10^'^ [0 which could be within the observable threshold of near future experiments. 
MSSM needs a new Higgs doublet and therefore its Higgs sector is phenomenologically richer. 
Then, one possibility for new physics is that the electroweak symmetry breaking involves 
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more that one Higgs doublet and without supersymmetry. In particular, if there is only 
one extra Higgs doublet, we are in the context of the so-called Two Higgs Doublet Model 
(2HDM). In such models, a new free parameter tan/3 arise from the breaking symmetry 
sector and as we shall see later, it could provide a significant enhancement for the FCN 
processes. There are three classes of these 2HDM which have been used to examine the 
ratios for rare top decays, called models I, II and HI. The first two are characterized by 
a discrete symmetry which avoid tree- level FCN processes @]. In the model type I, one 
Higgs Doublet provides masses to the up-type and down-type quarks, simultaneously. In 
the model type II, one Higgs doublet gives masses to the up-type quarks and the other one 
to the down-type quarks. In models type I and II the corresponding branching fractions 
for t — > c(7, Z, g) cannot approach in anyway the detectability threshold reaching only 3-4 
orders of magnitude above the SM values 0. 

But the discrete symmetry 0] is not compulsory and both doublets may generate the 
masses of the quarks of up-type and down-type simultaneously, in such case we are in the 
model type HI [1^]. It has been used to search for FCNC at the tree level In 
next sections we will show that in the context of the general 2HDM type HI, the tan/3 
enhancement might be enough to achieve the dectectability threshold. Recently, the 2HDM 
type HI has been discussed and classified |]TB|], depending on how the basis for the vacuum 
expectation values (VEV) are chosen and according to the way in which the flavor mixing 
matrices are rotated. In brief, the reference [T3\ considered two cases according to the VEV 
structure; in the case (a), both Higgs Doublets acquire a VEV, while in the case (b) only 
one of them get it. Further, there are two types of rotations which generate four different 
Lagrangians in the quark sector and two different ones in the leptonic sector. The well known 
2HDM types I and II, could be generated from them in the limit in which the mixing vertices 
at the tree level vanish. In that paper, has been pointed out that the phenomenology of the 
2HDM type HI is highly sensitive to the rotation used for the mixing matrices. Also it was 
shown that case (b) is a particular ocurrence of case (a). The model which includes all these 
features mentioned above, will be called from now on the General Two Higgs Doublet Model 
(G2HDM) type HI. The case (b) is the most familiar in the literature and calculations has 
been done for the branching fractions of top quark decays. In this particular version of 
model type HI values up to B{t cj) ~ 10"'^ , B{t cZ) ~ 10"^ and B{t eg) ~ 10~^ 



are gotten |TT 
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On the other hand, we must take into account the sensitivity of future experiments to 
these kind of processes and we must prepare the analysis of the phenomenology in different 
models in order to distinguish among them in case of a single event detection of this kind. Up 
to now, CDF has set limits on the FCN decays t — * c{g, Z, 7). In seeking t —>■ Zq they looked 
at pp ti qZbW, qZbZ ^ II + Ajets with high jet E^, taking into account the possible 
background from the SM reactions, they are getting the upper limit B{t — » cZ) < 0.33 
at 95% C.Lp|. To study t q'j, CDF has examined the reaction pp ^ tt ^ Wb'yq. 
If the W decayed leptonically, the signature would be 7 + / + Ej + (> 2) jets, but if it 
were hadronically, the signature would be 7 + (> 4) jets with one jet b-tagged. CDF set 
the 95 % C.L. upper bound B(t jq) < 0.032 0. Run II will provide much greater 
sensitivity to these decays, getting to explore ratios up to B{t —>■ q-y) ~ 8.4 x 10~^ and 
B{t Zq) ~ 6.3 X 10~^ at 100 fh^^ of the integrated luminosity 0. Other analysis 
for the sensitivity to the FCN decays of top quark have been made for the LHC collider 
0. For t —* Zq, ATLAS collaboration could achieve B(t — > Zq) ~ 2 X 10"^ at the 5a 
level and CMS collaboration up to B(t —>■ Zq) ~ 1.9 x 10""^ at the 5cr level, assuming an 
integrated luminosity of 100 fb^^. Now, for the decay t iq{u, c), ATLAS could go up to 
Bit 7g) ~ 10-^ while CMS about B{t 79) ~ 3.4 x 10"^ at 5(t level. 

Our aim is to calculate the branching ratio for t — 07 in the framework of the most 
general CP-conserving 2HDM type III, and it shall be shown that in this model is possible 
to reach the limits of the sensitivity expected in near future top quark experiments. This 



branching ratio was considered by Atwood, et. al. ||Tl[ in the 2HDM type III but they 
choose an specific set of values for the VEV, so that the parameter tan (3 does not appear in 
the analysis, which we call case (b). We have shown previously that the CP-conserving 
2IIDM type III with both VEV different from zero has two interesting features. First, it 
is possible to generate the Lagrangians for the models type I and II choosing conveniently 
the basis for the rotation of the mixing matrices in the Higgs sector. Second, in this model 
appears explicitly the parameter tan (3 which is the ratio of the vacuum expectation values. 
Such parameter plays an important role in the phenomenology of the model as we will show 
in next sections. In fact, this parameter appears as an important one in SUSY versions of 
the SM in a natural way, and there is not any fundamental reason to avoid its appearence in 
the 2HDMs type III. In order to give a better presentation we reviewed the general 2HDM 
type III in section II and then we present the analysis and concluding remarks for the FCN 
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decays of the top quark in section III. 



II. THE GENERAL 2HDM TYPE III 



The most general Yukawa's Lagrangian which includes a new Higgs doublet, is as follow 
-£y = V^,L^iU,R + r^f;Q,L^iD,R + vSIl^iE.r (1) 

where $1,2 are the Higgs doublets, rj^^ and are non-diagonal 3x3 matrices and i, j are 
family indices. 

In this kind of model (type III), we consider two cases. In the case (a) we assume the 
VEV as 

^ ^ \ , . i ' 
V1/V2) \V2/V2 

The mass eigenstates of the scalar fields are given in ref. The case (b) corresponds to 

the case in which one of the VEV is equal to zero, 

/ 



(<^>i)c 



(2) 
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[vi/V2 



, ('^2)0 = 0. 



(3) 



The mass eigenstate scalar fields in this case are given in ref. |10[- An important difference 
between both models is that the would-be Goldstone bosons Gz(w) are a linear combination 
of components of $1 and $2 in the model (a), meanwhile in the model (b) are a component 



of the doublet $1 13 



To convert the Lagrangian (0) into mass eigenstates we make the unitary transformations 
from which we obtain the mass matrices. In the context of 



M' 



diag 
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Vl 
Tl 



^1 -,u , 

71' +v/2 



R- 



(4) 



Where V^r) and Tl{r^ rotate down-type quarks and up-type quarks, respectively. 

We can solve for (rotations of type I). Replacing them into (|^) the expanded 

neutral Lagrangian for up and down sectors are 

^ -QM^'^'Q (sin aH" + cos 



r{a,I) 

■^Y{Q=U,D) 



2Mw sin P 



Q7]'^Q sin (a - (3) + cos (a - 13) 



zikZvy ^ v2 sm p 

- -mt'^^QGi (5) 



2MvK 

where r/^^-^) = TL(VL)?7^*^'^^'°Tjj(Vfi;)"'^ and similarly for The superindex (a,/) refers 

to the ) and rotation type I. It is easy to check that the Lagrangian constructed as 



^Y{u) + ^Y{D) is Lagrangian of the 2HDM type I ||T4[, plus FC interactions. Moreover,, 



it is observed that the case (b) in both up and down sectors can be calculated just taking 
the limit tan/3 — * oo. 

On the other hand, from (^, we can also solve for r^^'^, 7]^'*^ instead of ^^'^ which we 
call rotations of type II. Replacing them into (|^) the expanded neutral Lagrangian for up 
and down sectors become 

-£^yiilu,D) = 2mJcos/3 ^^q"'^^^"^^^° " ^^^"^"^ 

+ g^^g [sin(a - P)H^ + cos(a - P)h" 

\/2cos/3 '- 

2,Mw ^ V2 COS (3 

T ^QK'^'^^QGz- (6) 

In this situation the case (b) is obtained in the limit tan/3 — > 0. Moreover, the Lagrangian 



built cLS ^Y(Tj\ ~l~ "^YiD^ 

coincides with the one of the 2HDM type II |T^, plus some FC inter- 



actions. In addition, we can build two additional lagrangians by adding £y[u) + ^yId) 
•^¥{11) + -^yId)- different Lagrangians are generated from the case (a). Therefore, 

with the two Lagrangians (^) and (^) is possible to describe all possible CP conserving neu- 
tral current interactions in the quark sector of any model that includes only one extra Higgs 
doublet as Physics beyond the SM. 



III. THE RARE DECAY t 

In the present work, we calculate the branching fraction for the rate of the top quark 
decay t ^ using the model presented in section II. The widths of t ^ 07 for the rotations 
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type I and II are 



C7) 




n) 




where 



(8) 



with Cij{mD = Cij{P'^ = m^, 0, 0, m^, m^) the Veltman-Passarino integrals and the defini- 
tions m2 = m^/mf, r]'{i')ij = viOij/9- 

We have taken into account the contributions due to the diagonal and nondiagonal terms 



the FC couplings are proportional to the geometric average of the quark masses, ri{C,)ij ~ 



In figure 1, we have plotted the branching ratio B{t c-f) for the rotations type I and 
II. We set a = 0, rriH = mh and m^o high enough to have only one scalar mass varying in 
the plot. The plot shows the behaviour with tan/3 = 0.1 (dashed line) for rotation type I 
and tan /? = 15 (solid line) for rotation type II. We see that for reasonable values of tan f3 is 
possible to reach values in the range of the sensitiviy of future top quark experiments. This 
is because the expressions for the widths in rotation type I and II have a strong dependence 
on the tan/5 parameter in such way that in rotation type I we can enhance the branching 
ratio choosing tan j3 small and in rotation type II we can do it setting tan (3 large enough. 
We also point out that in the case of rotation type II with tan P = and a = we reproduce 
the results already obtained by Atwood, et.al. [PH]. 

In figure 2, we have plotted the branching ratio B(t ^ cy) versus a for rotation type 
II with niH = ruh = 150 GeV and tan/5 = 8. We can see that it is possible to get values 
of the order of the sensitivity expected by coming soon experiments in top decays ~ 10~^ 
with even lower values of tan (3. And, in fig. 3 we are showing the lower bounds of tan (3 in 
rotation type II according with the value of the scalar Higgs boson mass which are enough to 
reach the experimental discovery range, i.e. for B(t cj) = 10^^(10^^) solid line (dashed 
line) . 



of rj[^)^'^ but we parametrize them using the prescription of Cheng and Sher |T^, where 



g,/m~WT~/2mw- 



7 



In conclusion, we can observe that the bounds obtained in this paper are highly sen- 
sitive to the type of rotation. Also we might notice that in the 2HDMs type III tan/? 
should be taken into account due to its importance as a free parameter related with the 
symmetry breaking sector. Instead of that, we are showing that by driving tan/3 properly, 
it is possible to enhance the rare branching B{t cy). Further, it is worth to remark 
that this enhancement in 2HDM type III could also appear in other rare top quark decays 
t c{Z or g), to achieve the corresponding experimental thresholds. Specifically, for values 
of -B(t — s> C7) ~ 1 — 9 X 10^^ which are the orders of sensitivity of the experiments, we get 
the allowed region 8 ^ tan/3 ^ 15 for rotation type II. 
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FIG. 1: The branching ratio for t ^ cy for the rotation type I (dashed Hne) and rotation type II 
(solid Une). Setting a = 0, niH = fUh and ruA decoupled. For rotation type I (II), tan/3 = 0.1(15). 
The plot displays that for < 300 GeV, this branching ratio may get the sensitivity threshold 
for near future top quark experiments under this basic assumptions. 
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FIG. 2: The B{t cj) as a function of a in the context of rotation type II, for tan/3 = 8, 
ruH = rrih = 150 GeV and iua decoupled. The range for B{t 07) of 10~^ is reached with a lower 
value of tan 
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FIG. 3: Contourplot for the B{t 07) = 10"^ (10"^) solid line (dashed line) in the tan/3 - niH 
plane, for mn = ruh and a = 0. The range for B{t — 07) is the range of sensitivity of CMS, CDF, 
ATLAS experiments in near future. 
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